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(54) A computer user Interface using a manipulatory grammar 



(57) A method for inputting information to a device 
connected to a deformable piece includes the steps of 
manipulating the deformable piece to provide a first mor- 
pheme input to the device. The first morpheme input 
normally triggers a first default action by the device, 
such as controlling a display modifying a data structure, 
or communicating with another electronic device. When 
a user asynchronously manipulates the deformable 
piece to provide a second morpheme input to the device, 
the second morpheme input converts the normally trig- 
gered first default action to a second action generally 
not equivalent to the first default action. This mode of 
interaction allows formation of morphemic sentences to 
control a graspable device. 

The method is implemented by a device (10) sup- 
porting a manipulatable user interface, the device com- 



prising: 



a feedback module (33) for presenting information 
related to a data structure, 
a processor (24) lor controlling the feedback mod- 
ule (33) and the data structure, and 
a deformable piece (20) including multiple subre- 
gions. The deformable piece is attached in a vicinity 
to the feedback module (33) : with the deformable 
piece contacting at least one sensor (22) that mon- 
itors positional changes within multiple subregions 
of the deformable piece (20). The at least one sen- 
sor (22) is connected to the processor (24). 
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Description 

[0001] Reliably, quickly, and intuitively transmitting 
complex commands to small portable computers can be 
difficult. Small computer devices do not generally have 
sufficient computer processing power to respond relia- 
bly to voice or handwritten (pen based) commands, key- 
boards are often absent or too small for accurate finger 
input, and conventional buttons are too large or support 
too limited a command instruction set. User interface 
techniques that rely on bulky external modules (full size 
infrared linked keyboards, tethered data gloves, or cam- 
era based gestural recognition equipment, for example) 
are expensive, often not readily available outside select- 
ed sites, and probably too awkward for widespread use 
in conjunction with consumer level portable computing 
devices 

[0002] User interface designers for portable comput- 
ers have attempted to compensate for some of these 
problems by constructing devices that rely on various 
spatial, positional, or environmental cues that manually 
or automatically allow for activation of various modes in 
the device. For example, some laptop computers use 
the action of opening/closing the lid to initiate automatic 
bootup/powerdown of the computer without requiring 
any additional signal input (e.g. such as depressing a 
"start" button or typing "l_o_g_o_n" on a keyboard) from 
a user. Alternatively, the use of small portable comput- 
ers that automatically switch control modes depending 
on position, orientation, or room location have been in- 
vestigated. Button less manual control of a portable 
computer through deliberate user actions such as tilting 
the portable computer have also been described. 
[0003] However, all these solutions for interfacing with 
small portable computers have generally been limited 
in scope and functionality. What is needed is a user in- 
terface system suitable for even very small portable 
computers (having volumetric dimensions on the order 
of a one cubic centimeter) that is powerful, can be intu- 
itively operated by an ordinary user with little training, 
and is still readily capable of modification or extension 
by the user. The present invention meets these require- 
ments by providing a manipulatory user interface that 
responds to a user twisting, folding, bending, squeezing, 
shaking, tilting, spinning, lifting, or otherwise physically 
manipulating the computer. 

[0004] In the manipulatory user interface system of 
the present invention, the most basic level of manipula- 
tion is known as a "senseme". A senseme is defined as 
a single indivisible type of physical manipulation. A par- 
tial list of categories of sensemes include material trans- 
formations such as squeezing, twisting, stretching; local 
spatial transformations such as translation, rotation, or- 
biting; and environmental transformations based on 
temperature, light level, or vibration. For example, a 
small portable computer may support a deformable 
piece having multiple embedded sensors that detect 
folding, twisting, or bending of the deformable piece by 



a user. This computer can also contain a number of ac- 
celerometers that sense relative spatial information; gy- 
roscopic, radio or infrared positional sensors for deter- 
mining absolute position; and various thermal or photo- 

5 sensors that respectively detect temperature and light 
level changes. Intentional or unintentional modifications 
detected by one or more of these sensor systems can 
provide the basis for a powerful user interface scheme. 
[0005] As will be appreciated, each senseme catego- 

io ry contains many individually distinguishable members. 
For example, consider the category of senseme known 
as a "pinch", a structural transformation generally com- 
pleted by a user squeezing the deformable piece be- 
tween a forefinger and thumb. A pinch can be modified 

»5 by varying its speed (quick or slow pinch), magnitude/ 
intensity (light or hard pinch), portion of deformable 
piece pinched (top, bottom : or center of deformable 
piece pinched), or even portion of body used to pinch 
(right handed pinch or left handed pinch) : with each 

20 modification being distinguishable as a senseme capa- 
ble of being mapped onto a computer control command. 
[0006] Although the wide variety of easily distinguish- 
able sensemes would alone provide a powerful user in- 
terface to a computer, the present invention further ex- 

2S tends the flexibility of the senseme based user interface 
by supporting computer control based on a •morpheme- 
input. The morpheme is a temporally synchronous (or 
overlapping asynchronous) tuple of one or more sense- 
mes. Note that a morpheme can (and often will) contain 

30 more than one senseme. The sensemes combined into 
a morpheme can come either from the same category 
(the user pinches with a right hand while tapping with a 
left hand finger), or different categories (the user pinch- 
es the deformable piece with a right hand while modify- 

35 ing the spatial position of the portable computer by tilting 
it forward). 

[0007] Any morpheme can in turn be extended by par- 
ticipation in a "sentence". A sentence is defined as a 
sequence of one or more temporally disjoint mor- 
40 phemes. The sentence level allows definition of a phys- 
ical manipulatory grammar by appropriate choice of 
morpheme sequence, and corollary rules governing, for 
example, use of active (verb like) morphemes, naming 
(noun) morphemes, or connectors. Other possible 
45 grammar constructs used in sentences may include 
those based on "home" systems. Home systems are 
general-purpose gestural languages, whose grammar 
and syntax are not borrowed in any way from a host lan- 
guage. Examples ol these languages are gestural lan- 
50 guages developed by deaf children of hearing parents 
who have not been exposed to American Sign Lan- 
guage (ASL), and the "plains talk" of North American 
Indians, which was used as a trade language. 
[0008] Accordingly, a first aspect of the present inven- 
ts tion provides a method for inputting information to a 
computer connected to a deformable piece that can be 
manipulated, and optionally to various position sensors 
(both relative and absolute), pressure sensors, thermal 
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sensors : or even light sensors. The method comprises 
the steps of manipulating the deformable piece to pro- 
vide a first morpheme input to the computer, with the 
first morpheme input normally triggering a first default 
action by the computer. The deformable piece may also 5 
be manipulated to provide a second morpheme input to 
the computer, with the second morpheme input convert- 
ing the normally triggered first default action to a second 
action. The first and second morphemes (and any sub- 
sequent morphemes) together form a sentence that can 10 
be interpreted as a command to implement a computer 
controlled action, whether it be to unlock an electroni- 
cally controlled door, display a graphical image on a 
computer display or begin logging on to a computer net- 
work. Advantageously, such a user interface system is *5 
well suited for interaction with small computer devices, 
and may even be cross-cultural to a limited extent, with 
ability to squeeze or pinch being universal human at- 
tributes. 

[0009] According to a second aspect of the present 20 
invention there is provided a device supporting a manip- 
ulatable user interface, the device comprising: 

a feedback module for presenting information relat- 
ed to a data structure, 25 
a processor for control ling the feedback module and 
the data structure, and 

a deformable piece including multiple sub regions, 
the deformable piece attached in a vicinity to the 
feedback module, with the deformable piece con- 30 
tacting at least one sensor that monitors positional 
changes within multiple subregions of the deform- 
able piece, the at least one sensor being connected 
to the processor. 

35 

[0010] According to a third aspect ol the present in- 
vention there is provided a device supporting a manip- 
ulatabie user interface, the device comprising: 

a processor and memory, and 40 
a deformable piece including multiple subregions, 
the deformable piece attached in a vicinity to the 
processor and memory, with the deformable piece 
contacting a sensor mesh that monitors positional 
changes within multiple subregions of the deform- <*5 
able piece, the sensor mesh being connected to the 
processor. 

[001 1] According to a fourth aspect of the present in- 
vention there is provided a device supporting a manip- 5C 
ulatable user interface, the device comprising: 

a feedback module for presenting information relat- 
ed to a data structure, 

a processor lor controlling thef eedback module and ss 
the data structure, and 

a detector tor determining handedness of a user, 
the detector being connected to the processor for 



modifying the displayed data structure based on 
handedness of a user. 

[001 2] According to a fifth aspect of the present inven- 
tion there is provided a method for inputting information 
to a device connected to a deformable piece, the meth- 
od comprising the steps of : 

spatially rotating device to provide a first morpheme 
input to the device, the first morpheme input nor- 
mally triggering a first default action by the device, 
and 

asynchronously manipulating the deformable piece 
to provide a second morpheme input to the device, 
with the second morpheme input converting the 
normally triggered first default action to a second 
action. 

[0013] According to a sixth aspect of the present in- 
vention there is provided a plurality of tileable devices 
for transferring data, comprising: 

a first device having a processor and a first wireless 
communication module for transferring data, 
a second device having a processor and a second 
wireless communication module for transferring da- 
ta, 

a third device having a processor and a third wire- 
less communication module for transferring data : 
wherein the first device is connected in substantially 
simultaneous wireless communication with the sec- 
ond device and the third device to pass data based 
on spatial positions of the respective first device, 
second device and third device. 

[0014] The present invention is particularly useful for 
portable computers that can be held in the palm of a 
hand. If the portable computer is partially or completely 
surrounded with a deformable material having embed- 
ded or contact pressure/deformation sensors, a user is 
able to gesture with or manipulate the whole computer 
in order to achieve some desired result. Material defor- 
mation can be implemented at various scales. For in- 
stance a computer and connected deformation sensors 
mounted inside a flexible tube could respond to right an- 
gle bending of the tube, or even complex knotting or 
looping of the tube. In practice however, only minute sur- 
face deformations are required, being just sufficient to 
provide tactile feedback to pressing, pinching, or bend- 
ing type manipulations. In either case, however, the 
measurement of location and pressure applied to a sur- 
face is sufficient to characterize the mode of interaction 
(distinguishing a pinch from a prod). 
[0015] One particularly preferred embodiment of a 
handheld portable computer that responds to a physical 
manipulatory grammar in accordance with the present 
invention includes a computer, a feedback module to 
provide visual, auditory, or tactile feedback to a user (e. 
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g., processor connected LCD display, audio speaker, or 
tactile display to present Braille or other conventional 
touch interface), and co-mounted graspable, deforma- 
ble piece partially or completely surrounding the feed- 
back module. In addition, various thermal or pressure 
sensors are mounted to detect handedness of a user, 
with the grasping hand generally being the non -domi- 
nant hand for the user. Depending on the handedness 
ot the user (which can be considered as a morpheme), 
the displayed data structure is modified. For example, 
text displayed on an LCD display screen may be auto- 
matically shitted rightward on the screen to allow pen 
based annotation on the left side of the screen, thereby 
aiding left handed users. 

[001 6] Physically manipulatable user interfaces addi- 
tionally provide an opportunity for multiple devices to in- 
teract in a user friendly manner. For example, a tileable 
display system having multiple displays with touch sen- 
sitive deformable pieces surrounding the display can be 
used to organize documents based on relative position 
of contact of each display. For example, if two displays 
initially showing separate data structures (e.g two differ- 
ent pages from two different electronic books) are 
brought together in side by side touching contact, the 
displayed visual information can alter (e.g. by displaying 
adjacent pages of a single electronic book). As will be 
appreciated, one can also use multiple computers con- 
nected to physically manipulatable control elements to 
construct complex commands for organizing data struc- 
tures. 

[0017] The method of the present invention may fur- 
ther comprise the step of spatially manipulating the de- 
formable piece as at least one of the first and second 
morpheme inputs to the device. 
[001 8] At least one of the first and second morpheme 
inputs to the device may require spatially translating the 
device, panning the device, pushing the device relative 
to a user, pulling the device relative to a user, moving 
the device within a room and/or moving the device be- 
tween widely separated sites. 

[001 9] At least one of the first and second morpheme 
inputs to the device may be based on detected light var- 
iations, detected thermal variations, detected electro- 
magnetic variations, detected vibration variations and/ 
or detected acoustic variations. Information can be input 
to the device of the present invention by; 

a) shaking the deformable piece to provide a mor- 
pheme input to the device and triggering a first de- 
fault action by the device in response to shaking the 
deformable piece; 

b) flicking the deformable piece to provide a mor- 
pheme input to the device, and triggering a first de- 
fault action by the device in response to flicking the 
deformable piece; 

c) facing the deformable piece to provide a mor- 
pheme input to the device, and triggering a firs! de- 
fault action by the device in response to facing the 



deformable piece; 

d) lifting the deformable piece to provide a mor- 
pheme input to the device and triggering a first de- 
fault action by the device in response to lifting the 

5 deformable piece; 

e) orienting the deformable piece relative to an ex- 
ternal environmental object to provide a morpheme 
input to the device, and triggering a first default ac- 
tion by the device in response to orienting the de- 

10 formable piece relative to an external environmen- 
tal object; and/or 

f) whacking the deformable piece to provide a mor- 
pheme input to the device, and triggering a first de- 
fault action by the device in response to whacking 

is ihe deformable piece. 

[0020] Embodiments of the present invention will now 
be described, by way of example, with reference to the 
accompanying drawings, in which: 

20 

Figure 1 is a schematic diagram of an approximate- 
ly spherical hand holdable portable computer hav- 
ing a deformable surface, a status display, and a 
pressure sensor array for detecting surface defor- 
ms mations; 

Figure 2 is a graphical diagram illustrating various 
physical manipulation morphemes, with axes re- 
spectively illustrating complexity of senseme tuples 
required to form a morpheme, and plasticity of a de- 
30 vice required to support the physical manipulation; 
Figures 3-16 schematically illustrate various pre- 
ferred classes of physical manipulation mor- 
phemes; 

Figure 17 is a graphical diagram illustrating various 
35 spatial morphemes, with a first axis illustrating com- 
plexity of senseme tuples required to form a mor- 
pheme, and a second axis illustrating the degree of 
positional information needed to support the phys- 
ical manipulation (moving along a continuum from 
40 relative local measurement along a single specified 
dimension to absolute global measurement with six 
degrees ol freedom determined); 
Figures 18-26 schematically illustrate various pre- 
ferred spatial manipulation morphemes; 
45 Figure 27 is a graphical illustration showing increas- 
ing degrees of sensor system complexity that can 
be used to detect various categories of environmen - 
tal stimuli, including light effects, thermal effects, 
Ihe electromagnetic environment, and the vibratory/' 
50 acoustic environment; 

Figure 28 is a graphical diagram illustrating various 
physical manipulation morphemes for multiple in- 
teracting devices, with axes respectively illustrating 
the complexity ol senseme tuples required to form 
55 a morpheme, and increasing level of physical con- 
tact; 

Figures 29-34 schematically illustrate various pre- 
ferred manipulation morphemes for multiple inter- 
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acting devices; 

Figures 35 and 36 are schematic diagrams illustrat- 
ing "squeeze" and "tilt" morphemes applicable to a 
portable computer; 

Figure 37 is a schematic diagram illustrating tilt and 5 
squeeze morphemes used to control view ol large 
two dimensional data sets with a relatively small dis- 
play of a portable computer: 
Figure 38 is a schematic illustration representing a 
portable computer having a display ready to receive ^ 
annotations from a right handed user; 
Figure 39 is a schematic illustration representing a 
portable computer having a display ready to receive 
annotations from a left handed user; 
Figure 40 is an electronic schematic illustrating 15 
components of the pressure and tilt sensitive mod- 
ules of a portable computer such as illustrated in 
Figures 35-39. 

Figures 41 and 42 are schematic illustrations of a 
scanner/printer/copier using a paper shaped dis- 20 
play interface to support morphemic input; 
Figures 43-45 are schematic illustrations of tileable 
displays capable of supporting morphemic input; 
Figure 46 illustrates optical sensors and patterns 
suitable for use in conjunction with tileable displays 25 
such as illustrated in Figures 43-45 ; 
Figure 47 illustrates radio transponders suitable for 
use in conjunction with tileable displays such as il- 
lustrated in Figures 43-45; and 

Figure 48 illustrates addressing of multiple tileable 30 
displays. 

[0021] Figure 1 illustrates an embodiment ol the 
present invention suitable for supporting a morphemic 
user interface grammar. Support of the grammar can re- 35 
quire detection of a user's physical manipulation of a de- 
vice, detection of relative or absolute spatial location of 
the device, detection of various environmental factors 
acting on the device, and even detection and interaction 
with multiple devices or external computer networks. As 40 
illustrated, a device 1 0 has a deformable surface 20 with 
an underlying deformation sensor mesh 22 for detecting 
surface deformation across or within multiple subre- 
gions of the deformable surface 20. The deformation 
sensor mesh 22 is connected to an internally contained 45 
processor 24 having associated memory system 26. For 
detecting various positional or environmental variables, 
a sensing system 28 is also provided. The illustrated de- 
vice further includes a feedback module 33, which may 
include an externally visible status display 30 or a non- so 
visual feedback module 31 (typically delivering auditory 
or tactile feedback). In the illustrated device, a commu- 
nications system 32 for reception or transmission of in- 
formation to other electronic or computing devices is al- 
so provided. All these components can be powered by 55 
a power supply 25, which is usually an internally mount- 
ed rechargeable battery of conventional construction. 
[0022] Although the device 1 0 is illustrated as having 



an approximately spheroidal and unitary mass, various 
other shapes are contemplated to be within the scope 
of the present invention. For example, the overall shape 
may be similar to various rectangular prisms, or can be 
ellipsoidal, toroidal planar, or even be malleable 
enough to support a wide range of user defined irregular 
shapes. In addition, multiple cooperating shape ele- 
ments are contemplated using conventional designs 
that permit interlocking of multiple shape elements (e.g 
using a ball and socket, a lock and key, or slidable or 
rotatable interlocked components). 
[0023] Whatever the shape of device 1 0, for operation 
of the present invention the device 10 is completely or 
partially enveloped by the deformable surface 20. The 
present invention supports use of a great variety of de- 
signs and materials for the deformable surface 20, de- 
pending on the required plasticity, durability longevity, 
and of course, cost constraints. For example, contem- 
plated designs for deformable surface 20 include, but 
are not limited to: 

a closed or open celled polymeric foam material 
having a wall thickness of millimeters to centime- 
ters, with thinner walled embodiments being sup- 
ported (e.g. by adhesive attachment) by an internal 
hard shell (constructed from polymeric or metallic 
materials), and those thicker walled embodiments 
directly supporting (by, e.g. brackets or supports) 
internal components such as processor 24. Suita- 
ble foams may include those composed in whole or 
in part of widely available synthetic rubbers such as 
polychloroprene (neoprene), polystyrenes, rubber 
or nitrile rubber latex foams, polysiloxanes, block 
polymers including styrene-butadiene or styrene 
isoprene, or any other conventional material having 
good elasticity and deformability; 
a thin single layer polymeric surface loosely 
wrapped around a internal hard shell (the hard shell 
being constructed from polymeric or metallic mate- 
rials). For example, a nylon or cotton weave, single 
layer polyethylene, synthetic rubber (with little or no 
foam cells present), or natural polymeric materials 
such as leather wrapped around a polystyrene cas- 
ing can be used; 

a composite layered surface having a durable pol- 
ymeric outer layer supported by an inner foam layer; 
or even 

a polymeric bilayer having an intermediate fluid or 
gel layer of a viscous or thixotropic material that can 
be used to support extreme deformations. The in- 
termediate layer can be relatively thick (one the or- 
der of centimeters), or in certain embodiments can 
have a thickness measured on micron to millimeter 
scales. Such extremely thin layers would allow 
complex twisting, folding, curling, or crumpling ac- 
tions, and have been described in conjunction with 
U.S. Patent 5,389,945, assigned to Xerox Corp., 
the disclosure of which is herein specifically incor- 
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porated by reference. 

[0024] The deformation sensor mesh 22 can be em- 
bedded within, or positioned to contact, the deformable 
surface 20. The deformation sensor mesh 22 can in- 5 
elude an array of individual compressional or tensional 
strain sensors, or alternatively, embedded or attached 
positional sensors. For certain applications, continuous 
sensors (e.g. bilayer sheets of capacitance sensors) 
may be employed. One particularly useful continuous 
sensor type uses multiple capacitance or resistance 
strips, with deformation pressure resulting in a position- 
ally localizable analog signal proportional to the applied 
deformation pressure. Various sensor types can be 
used, including simple capacitance sensors, resistive 
strain sensors, analog or digital pressure switches, in- 
ductive sensors, or even fluid flow sensors. Depending 
on the sensor type employed, sensor data can be direct- 
ly fed to the processor 24 in digital forrrv or be trans- 
formed to digital format by an general purpose analog/ 
digital converter that typically provides a 4 or 8 bit range 
(although as few as one or as many as 32 bits may be 
required by various applications). An analog to digital 
converter may be internal to the processor 24 or provid- 
ed as an external module. As will be appreciated, the 
sensor mesh 22 is intended to include combinations of 
sensors and sensor types, which can be used over the 
whole or part of the deformable surface 20. 
[0025] A positional or environmental sensor system 
28 can also be supported by device 10. Various sensor 
modes can be supported, including absolute or relative 
positional information as determined by gyroscopic sen- 
sors, accelerometers, or acoustic or infrared ranging 
techniques. Environmental sensors : including conven- 
tional light, image, thermal, electromagnetic, vibratory, 
or acoustic sensors can also be present. Depending on 
the desired application, even costly environmental or 
positional sensors such as those incorporating differen- 
tial GPS positioning, image analysis or recognition, 
acoustic or voice identification, or differential thermal 
sensors can be used as morphemic input. Such mor- 
phemic input, taken in conjunction with morphemic input 
as detected by sensor mesh 22. can enhance precision 
and flexibility of a user's control of device 10. 
[0026] As illustrated, both sensor system 28 and sen- 
sor mesh 22 are connected to the processor 24 and as- 
sociated memory 26. The processor 24 and memory 26 
are typically mounted within the deformable surface 20, 
by either direct attachment to the deformable surface 20 
or by attachment to a hard casing positioned within the 
deformable surface 20. Conventional CISC or RISC 
processors can be used in the illustrated embodiment, 
with low power processors such as the Signetics 87c752 
or 87c751 , Motorola 68HC1 1 or 685B2, or ARM 71 0 be- 
ing preferred. If convenient, coprocessors such as ana- 
log to digital converters or digital signal processors can 
be used alone or in conjunction with a main processor. 
Conventional flash, static, or dynamic RAM can used in 



the present invention, although for certain applications 
higher cost embedded DRAM may also be used. In 
some storage intensive applications, memory 26 can in- 
clude additional harddisk storage, either located within 
the device 10 or available through an external connec- 
tion. As will be appreciated, for many applications use 
of optional external communications can at least partial- 
ly supplant use of internal processors and memory (ex- 
cept for that necessary to support required sensor or 
communication buffering and signalling). 
[0027] The present invention optionally supports 
communications with an external computer system 40 
using its internal communications system 32 and asso- 
ciated transceiver 34. The external computer system 40 
also includes a transceiver 42, a personal computer or 
workstation 44, and is connected to a local or wide area 
network computer system 46. The transceivers 34 and 
42 can support various communication protocols and 
designs, including use of a serial tethered line 36 (using, 
for example the RS-232C interlace protocols), use of in- 
frared signals 38 adhering to widely utilized IRDA com- 
munication standards, or use of radiofrequency signal 
37 (which can be : for example, a cellular telephone, 
900MHz radio, or digital PCS telephonic communica- 
tions). Alternative communication standards, or even al- 
ternative communication carriers such as those based 
on optical or acoustic techniques, can of course be em- 
ployed. 

[0028] As will be appreciated, in addition to direct 
communication with external computer system 40, the 
device 1 0 can be directly or indirectly maintained in con- 
tinuous or intermittent communication with a number of 
suitably equipped electronic devices, including a tablet 
computer 110. or even a physically manipulatable port- 
able computer 1 1 similar in design and function to device 
10. Communication can be direct to a target device, or 
through an intermediary retransmitler such as computer 
system 40. Other possible communication targets in- 
clude automation control systems, security authoriza- 
tion units, personal digital assistants, notebook comput- 
ers, or any other suitably equipped electronic system. 
[0029] Results of communications with external de- 
vices, presentation of device stored information, or de- 
vice status updates can all be provided to a user through 
processor 24 controlled update of feedback module 33. 
Feedback to a user can be primarily visual, such as can 
occur in conjunction with visual display 30. Generally, 
the display 30 can be a conventional passive or active 
matrix liquid crystal display, although use ol more so- 
phisticated (and expensive) displays based on various 
electrooptical or micromechanical techniques can of 
course be used. In addition, for certain devices a non- 
imaging display such as may be formed by a small 
number of status lights (e.g. red or green LEDs), or lo- 
calized or distributed chromatic changes (in conjunction 
with a deformable surface 22 constructed with suitable 
elect rochromic materials) may be all that is necessary 
for visual feedback to the user. 
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[0030] In some embodiments of the invention, visual 
output through display 30 may be augmented (or even 
replaced) with a non-visual display 31 . The non-visual 
display 31 can include tactile displays based on internal 
actuators, auditory feedback, or even displays based on 
conformal changes in device appearance. For example, 
one possible feedback display is based on internal au- 
ditory speakers (emitting a range of sounds from simple 
"beeps" to well formed speech, depending on available 
processor speed and functionality) for providing user 
feedback. As will be appreciated, non-visual display 31 
and its associated actuators or electronics can support 
alternative feedback modes, including, for example, 
force feedback to a user through internal actuators, tac- 
tile based feedback (e.g with multiple surface projec- 
tions for presentation of Braille or other conventional 
tactile user interface), modifications to the surface tex- 
ture of the device, or any other conventional mechanism 
for supplying information to a user. 
[0031] To better appreciate operation of the present 
invention, some selected modes of physical manipula- 
tion of device 10 are schematically presented in Figure 
1 . As seen in Figure 1 , device 10 can be translationally 
moved in three spatial dimensions, as illustrated with 
reference to orthogonal force arrows 50, 51, and 52. In 
addition to translational motion, the device 10 can be 
rotationally moved in any or all three spatial dimensions, 
as represented by arrows 53, 54 : and 55. By use of sen- 
sor system 28 (alone or in combination with communi- 
cation system 32), the relative or absolute position and 
orientation in three dimensions can be determined. 
[0032] In addition to determining spatial position and 
orientation through use of sensors 28, the device 1 0 can 
optionally use sensor mesh 22 to measure and localize 
transient or continuing force application, with force vec- 
tors and related timing information being determined 
and interpreted. A few possible force actions (deforma- 
tion modes) are schematically illustrated in Figure 1, 
with arrows 60 and 61 illustrating depression of surface 
20 (with the combination representing a squeeze), ar- 
rows 62 and 63 illustrating sliding or rubbing deforma- 
tion (with the combination representing a twist), and the 
combination ol sliding arrows 65 and 66, and outward 
pull 67, together representing a pinch and outward pull. 
The strength of the applied force can be measured (e. 
g. a hard or soft squeeze is differentiated), its spatial 
expanse found (e.g. to differentiate between poking with 
a fingertip or a thumb pad), and timing determined (e.g. 
to differentiate between a quick or a slow depression of 
the surface). The deformation so caused may be either 
permanent or transitory. 

[0033] As will be appreciated by those skilled in the 
art, each of the foregoing force actions represented by 
arrows can be considered a senseme. Some temporally 
distinguishable sensemes (or combinations of sense- 
mes such as the foregoing discussed pinch/pull combi- 
nation) further represent morphemes used as a basis 
for a morphemic grammar in accordance with the 



present invention. All of the following described mor- 
phemes can be modified by numerous variations in ap- 
plied pressure, force utilized, appendages used, body 
parts, or extraneous intermediary objects used to apply 

s force. In addition, timing of various objects (whether 
quick, slow, or alternately quick and slow) can modify 
interpretation of a morpheme. For example, if "squeeze" 
is taken as a typical morpheme, one can appreciate var- 
ious squeeze operations such as quick squeeze, slow 

10 squeeze, hard squeeze, soft squeeze, narrow squeeze, 
wide squeeze, squeeze between hands, squeeze be- 
tween one hand and a user's chest or head, squeeze 
between one hand and a table or wall : a squeeze made 
between two pens or two books, or even a squeeze be- 

15 tween a user's tongue and the roof of the mouth. For 
purposes of the present invention, all such squeeze 
morphemes would be considered members of the 
"squeeze" class, with individual variations acting as pos- 
sible modifiers or selected cases, just as a "house cat" 

20 might be considered a particular member of the class of 
"felines", which more generally embraces lions, tigers, 
and bobcats. 

[0034] To aid in understanding the diversity of con- 
templated physical manipulation morphemes, Figure 2 

25 illustrates selected morphemes arranged by increasing 
plasticity of the device required to enable morpheme uti- 
lization, and by increasing complexity of available 
senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 

30 with the least plastic device and the simplest senseme 
set used to compose a morpheme, a definition of a pos- 
sible physical manipulation and typical function invoked 
by that manipulation of a device similar (but o1 course 
possibly more complex) to that described in conjunction 

35 with Figure 1 is presented: 

DEPRESS 

[0035] Definition: Indenting one or more subregions 
40 of the device through the application of pressure. 

[0036] Example: As seen in Figure 3, consider a de- 
vice 122 having a display 123. The device 122 supports 
a graphical illustration software application which allows 
users to position geometric shapes or user defined 
45 graphical objects. The device can have tour pads 124 
placed around it one on each side. By deforming a par- 
ticular side, the user indicates a desire to "nudge" the 
currently selected geometric shape 125 away from that 
side to a new position 1 26. 

50 

SQUEEZE 

[0037] Definition: To deform one or more subregions 
of a device through the application of vectors of force, 
55 some components of which are directed towards each 
other, which compress the morphological structure of 
the device. This is a special case of depression. 
[0038] Example: As illustrated in Figure 4, consider a 
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device 132 which can display one or more documents 
in either iconic (closed) or textual display (open) modes. 
By selecting an open document 135, and then squeez- 
ing a deformable edge 1 34 of the device 1 32, the user 
indicates a desire to make the document "smaller', in 
this case to iconize it as icon 1 36. 

FOLD 

[0039] Definition: To deform a second subregion by 
bending the first subregion such that it partially or com- 
pletely overlaps the second subregion. Further defor- 
mations can be applied to that new morphological struc- 
ture on other subregions. 

[0040] Example: As illustrated in Figure 5, consider a 
device 142 which can display documents. Suppose this 
device 142 has been augmented such that a deformable 
horizontal "flap" 144 on atop edge of the device 142 can 
be folded to partially obscure a display 143. When the 
user makes this folding gesture, the user indicates a de- 
sire to password-protect ("hide") the currently displayed 
document. 

CURL 

[0041] Definition: Deforming one or more subregions 
of the device by spirally configuring subregions relative 
to each other in a cylindrical or circular morphological 
structure. 

[0042] Example: As illustrated in Figure 6, consider a 
device 150 which can display documents in various lan- 
guages (English, French, etc.). When the user takes 
such a device 1 50, curls in a direction indicated by arrow 
157 into a tube, and then uncurls it, this "abra-cadabra" 
gesture tells the device 150 to display the current doc- 
ument in a different language. 

STRETCH 

[0043] Definition: Deforming one or more subregions 
of a device through the application of vectors of forces, 
some components of which are directed away from each 
other, the vectors being applied at opposite ends of the 
device. Example: As illustrated in Figure 7 : consider a 
device 160 with a graphical software application which 
allows users to manipulate geometric shapes. By 
stretching the device 160, the user indicates a desire to 
"resize" or "rescale" the currently displayed shape 165 
to a larger size 1 66, the amount of resizing being a func- 
tion of the amount of deformation. Note that SQUEEZ- 
ING can indicate resizing the currently displayed to a 
smaller size. 

PINCH 

[0044] Definition: Manipulating one or more subre- 
gions by applying vectors of forces, aligned directly to- 
wards each other, on opposite sides of the affected su- 



bregion(s). This is typically, but not exclusively, accom- 
plished using two-finger tactile force. PINCH is a special 
case of SQUEEZE. 

[0045] Example: As illustrated in Figure 8, consider a 
5 device 170 which can copy documents. By performing 
a "pinching" action 1 75, the user indicates that they wish 
the next set of copies to be issued in stapled form. 

DOGEAR 

10 

[0046] Definition: Deforming a second subregion by 
folding a first subregion at a logical corner or edge of 
the second subregion, indicating a marker or location 
point (e.g. a bookmark) to be later referenced. DOGEAR 

is is a special case of FOLD. 

[0047] Example: As illustrated in Figure 9, consider a 
device 180 which displays a subset of the pages from a 
multi-page document. By "dogearing" the upper right 
corner 185 of device 180, the user indicates that they 

20 wish a bookmark associated with the currently displayed 
page or pages. 

TWIST 

25 [0048] Definition: Deforming one or more subregions 
• of the device through the application of two opposing 
rotational forces offset from each other by a non-zero 
difference about some central axis. 
[0049] Example: As illustrated in Figure 10, consider 
30 a device 190 which, over time, becomes degraded in 
some aspect of its performance (its disk becomes frag- 
mented, its memory needs garbage-collection, etc.). By 
performing a "TWIST" gesture 195, the user indicates 
that they wish the device to "wring itself out", performing, 
35 for example, garbage collection. 

RELIEF-MAP 

[0050] Definition: Deforming one or more subregions 
40 of the device by raising and or lowering them by either 
spatial transformation or the addition/removal of mate- 
rial. 

[0051] Example: As illustrated in Figure 11, consider 
a device 200 which can display documents in either one- 
45 or two- page format. When the user "scores" the device 
by making a vertical indentation about the center axis of 
the device while it displays a single page 206, the device 
200 interprets the morpheme to request display of doc- 
uments in a two-page format as pages 207 and 208. 

50 

RIP 

[0052] Definition: Deforming one or more subregions 
of the device by introducing a spatial discontinuity, by 
55 applying vectors of force to partially or totally disconnect 
these subregions from the device. 
[0053] Example: As illustrated in Figure 12, consider 
a device 210 which can copy some or all of its informa- 
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tion. When the user performs the "ripping" gesture 21 5, 
removing one or more subregions, the device 210 cop- 
ies its currently selected data set onto those subregions 
216 and 217. 

PERFORATE 

[0054] Definition: Deforming one or more subregions 
of the device by means of introducing a change in the 
spatial connectivity of the subregions such that a hole 
is introduced (either temporarily or permanently) in the 
device. 

[0055] Example: As illustrated in Figure 1 3, consider 
a device 220 which is used to route messages between 
various parts of a device network, and which displays 
this functionality to the user by means of lines 222 rep- 
resenting message pathways. When the user perforates 
the device 220 with a finger or object 224 to introduce 
a hole in one of these pathways 222, the system stops 
routing messages along that pathway. 

SIMILARITY 

[0056] Definition: Deformation of one or more subre- 
gions of a device previously configured to represent 
some other predefined object. Typically the device acts 
in a manner consistent with the behavior of the real- 
world object when it is manipulated in this manner. 
[0057] Example: As illustrated in Figure 14, consider 
a device 230 which contains text-to-speech and audio 
input capability and which presents itself to the user in 
an anatomically accurate shape of a human head. When 
the user opens lips 232 on the human head, internal 
sensors detect opening of the lips and activate text-to- 
speech capability. 

3D MAP 

[0058] Definition: The morphing of a device that can 
be molded around an external object to permit a sensor 
mesh to determine simu ttaneousfy size and shape of the 
external object. The range of possible external objects 
is large but is limited by the size of the device's solid 
inner-housing and the volume of the moldable material 
in the outer housing. In this system the device has the 
ability to accurately sense the amount of material from 
its inner surface to the outer edge of the moldable ma- 
terial (example: via ultrasonic sounding, similar to sonar 
in water) thus determining an accurate electronic model 
for the shape of the molded enclosure. 
[0059] Example: As illustrated in Figure 15 : by press- 
ing a device attached moldable material 244 around the 
surface of an external object (e.g. a cog 242) a device 
240 can automatically generate a CAD model of that ob- 
ject and store it in its memory. 



MIMICRY 

[0060] Definition: Deforming one or more subregions 
of the device such that the resultant morphological 
5 structure resembles a known real-world object and 
through this association of subregions, the device acts 
in a manner consistent with the object it resembles. 
[0061] Example: As illustrated in Figure 16, consider 
a device 250 containing a computer which contains text- 
10 to-speech and audio input capability, and which 
presents itself to the user as a misshapen blob having 
the consistency and plasticity of moldable putty or clay. 
When the user performs the "MIMICRY" action by mold- 
ing part of the device 250 to resemble an ear the audio 
15 input capability is activated. 

[0062] In addition to morphemes based on physical 
manipulation, various morphemes based on varying de- 
grees of relative or absolute spatial positioning are con- 
templated to be useful in practice of the present inven- 
20 tion. To aid in understanding the diversity of contemplat- 
ed spatial morphemes, Figure 17 illustrates selected 
spatial morphemes arranged by increasing knowledge 
of spatial position required to enable morpheme utiliza- 
tion, and by increasing complexity of available senseme 
25 tuples required to form or interpret a morpheme applied 
to a particular class of device. Beginning with a device 
having only rudimentary relative positioning functional- 
ity for supporting simple spatial sensemes, and ending 
with a device absolutely positionable to within centime- 
30 ters anywhere on Earth, a definition of a possible spatial 
manipulation and typical function invoked by that ma- 
nipulation of a device similar (but of course possibly 
more complex) to that described in conjunction with Fig- 
ure 1 is presented: 

35 

TRANSLATE (relative to device) 

[0063] Definition: The linear movement of a device's 
canter o1 mass from one position in space to another 

40 [0064] Example: Used to substitute for mouse control- 
led graphical "sliders' in conventional graphical user in- 
terfaces. When it is only physically possible to display a 
small amount of list, large listings can still be searched 
by "scrolling" a display window in response to the 

45 TRANSLATE morpheme. 

SHAKE 

[0065] Definition: Spatially translating all subregions 
so of a device by repeated movement in opposing direc- 
tions, such that the net translation is negligible. 
[0066] Example: As illustrated in Figure 18, consider 
a device 260 which is used as a calculating device. 
When the user performs the "SHAKE" gesture, the de- 
55 vice 260 clears its accumulator. 
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REVOLVE 

[0067] Definition: Rotating all subregions of a device 
by rotating the subregions about a point interna! to the 
device, about any arbitrary plane. 
[0068] Example: As illustrated in Figure 19, consider 
a device 270 which displays an imaged slice of volumet- 
ric data, such as medical data from a series of CAT 
scans. By rotating the device about a center point 272 
internal to the device 270 to a new position 274, the 
plane specifying the imaged slice is changed according- 
ly. 

TILT 

[0069] Definition: Rotating one or more subregions of 
a device by rotating the subregions such that one or 
more components of the rotary force are in the direction 
of gravity and the amount o1 rotation is between about 
-180 degrees and +180 degrees. 
[0070] Example: Consider a device which displays 
frames from an animation sequence on the side facing 
the user. As the device is tilted away, the speed of the 
animation increases - as its tilted towards the user, the 
speed of the animation decreases, analogous to oper- 
ation of a gas pedal. 

FLICK 

[0071] Definition: A forwards TILT immediately fol- 
lowed by an opposing backwards TILT 
[0072] Example: As illustrated in Figure 20, consider 
a device 2B0 which can transmit some subset of its data 
to another device. When the user performs the "FLICK" 
gesture by quickly tilting in the direction of arrow 282, 
followed by a reverse tilt along arrow 282, the device 
280 performs this transmission, towards the device (not 
shown) pointed to by the ray of the gesture. 

SPIN 

[0073] Definition: Rotating one or more subregions of 
the device by rotating the subregions about a point in- 
ternal to the device., such that the plane of rotation is 
one of the device's surface planes. SPIN can be consid- 
ered a special case of REVOLVE. 
[0074] Example: As illustrated in Figure 21 , consider 
a device 290 which can display a frame of video 295 
from a video sequence. The user performs the "SPIN" 
gesture in a counter-clockwise direction, the device dis- 
plays earlier frames in the sequence; when the gesture 
is performed in a clockwise direction (arrow 292) : the 
device 290 displays a later frame 296 in the sequence 
(represented by film strip 294). 

ORIENT 

[0075] Definition: Rotating one or more subregions of 



the device by rotating the subregions about the center 
of the device, such that the plane of rotation is one of 
the device's surface planes, and the amount of the ro- 
tation is a multiple of 90 degrees (i.e. to rotate the device 
s between cardinal compass points). Orient can be con- 
sidered a special case of SPIN, which in turn is a special 
case of REVOLVE. 

[0076] Example: As illustrated in Figure 22, consider 
a device 300 which can display a document in either 1 

w page, 2-page, or 4-page format ("1-up", "2-up" : or 
"4-up"). When the user performs the orient gesture in a 
clockwise direction (arrow 302), the device 300 increas- 
es the number of document pages it is displaying from 
one page 305 to two pages 306 and 307. Further orient 

'5 gestures would increase the number of displayed pag- 
es. When performed in a counter-clockwise direction, 
the device 300 decreases the number of pages it is dis- 
playing. 

20 FACE 

[0077] Definition: Manipulating one or more subre- 
gions of the device such that a first set of subregions is 
no longer bottom-most and a second distinct set of su- 
*5 bregions now assumes the first subregions former po- 
sition. 

[0078] Example: As illustrated in Figure 23, consider 
a device 310 which displays documents, and which al- 
lows users to edit such documents. Consider further the 
30 case in which the device presents itself to the user in 
the form of a cube, in which 6 different documents are 
displayed on the 6 different faces. When the user per- 
forms the "FACE" gesture by making a particular face 
top-most, the document which is now on the top-most 
35 face becomes editable by the user, while the document 
which is no longer top-most is no longer editable. 

LIFT 

40 [0079] Definition: The movement of a device's center 
of mass in a direction opposite to the current gravitation 
force acting on the device. 

[0080] Example: Commanding the device to display 
the computer's file system at a position one level higher 
45 jn the hierarchy. 

PAN 

[0081] Definition: The application of a TRANSLATION 
so to a device such that it is moved parallel to the front of 
the user's body, at a substantially constant height. 
[0082] Example: Viewing a spread-sheet on a device 
with a display so small only one cell can be shown. By 
PANing the device, the contents of the current row can 
55 be shown in sequence depending on the rate or amount 
of PAN. However, if the device were rotated away from 
its current orientation during the PAN, a new row would 
be chosen. The selection of the row could be dependent 
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on the deviation from the original orientation. 
PUSH-PULL 

[0083] Definition: Manipulating one or more subre- 
gions of the device by spatially translating them such 
that they are moved along a line of projection from the 
center of the device to the vertical axis of the user's 
body. 

[0084] Example: Consider a device with audio output 
capability. As the device is ■pushed" further away from 
the body, its audio output level increases. When it's 
"pulled" towards the body, its level decreases. 

WHACK 

[0085] Definition: The application of an accelerative 
or de-accelerative force to one or more subregions of a 
device such that said subregions contact or are contact- 
ed by an external object, causing an equal and opposite 
countering force. 

[0086] Example: As illustrated in Figure 24, consider 
a device 320 which can perform long and unpredictable 
database searches. When the user performs the 
WHACK gesture (e.g. upon a table 322), the current 
search is aborted. 

ORIENT (relative to environment) 

[0087] Definition: Manipulating two subregions of the 
device such that the line drawn between the centers of 
those two subregions alters its orientation with respect 
to the surrounding environment. 
[0088] Example: Displaying a CAD drawing of a ma- 
chine-part in 3D on the display of a mobile device. As 
the orientation of the device changes, so too does the 
viewing angle and position of the rendered image. 

ORBIT 

[0089] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 
device about some point exterior to the physical bound- 
aries of the device, about any arbitrary axis. 
[0090] Example: As illustrated in Figure 25, consider 
a device 330 which can search a network database, 
such as the World Wide Web, for information. When the 
user perlorms the "orbit" gesture, such a search is initi- 
aled. The radius 332 of the rotation 334 specifies the 
breadth of the search - wider circles specify a wider 
search The speed of the gesture specifies the time limit 
imposed on the search - the quicker the gesture, the 
more cursory the search. 

ORBIT RELATIVE TO USER 

[0091] Definition: Rotating one or more subregions by 
rotating the subregions and/or the center of mass of the 



device about some point exterior to the physical bound- 
aries of the device, where said point is proximal to a 
body feature of the user. This is a special case of ORBIT 
[0092] Example: As illustrated in Figure 26, consider 
5 a device 340 which can perform audio output. By per- 
forming the ORBIT gesture (in direction indicated by ar- 
row 344) about the user's ear 345, the audio output is 
activated. 

10 MOVEMENT IN ROOM 

[0093] Definition: The local detection of a device's 3D 
position relative to reference points found within an en- 
closing room. Differences in the measured position are 
is used to trigger actions. 

[0094] Example: A virtual filing system that allows you 
to save and restore files based on the devices current 
position in the room. To save a file you might think care- 
fully about the contents of the file and then walk to the 
20 position in the room that might be most easily associated 
with it. When retrieving the file you would use the same 
thought processes and go back to the position you had 
associated with the file. On doing so the files associated 
with that position would be displayed and you would 
25 then be able to select the file you were looking for. The 
system is useful because the human mind is very good 
at remembering information that is spatially organized 
rather than in some abstract information data structure. 



30 MOVEMENT BETWEEN WIDELY SEPARATED SITES 

[0095] Definition: Manipulating one or subregions of 
the device such that the delected absolute spatial posi- 
tion of those subregions is changed. 
35 [0096] Example: Consider a device which can display 
information from a database of client information. When 
the device is moved to a different client site, the device 
automatically updates its display to display information 
for the nearest client site. 

[0097] In addition to morphemes based on physical 
manipulation or spatial positioning, various morphemes 
based on sensed environmental conditions are contem- 
plated to be useful in practice of the present invention. 
To aid in understanding the diversity of contemplated 
environmental morphemes, Figure 27 illustrates select- 
ed environmental morphemes loosely arranged in order 
of increasing sensor complexity needed in some com- 
monly sensed environmental categories. For each of the 
proffered categories, some selected sensing syslems 
supportable by a device such as that described in con- 
junction with Figure 1 are presented: 

LIGHT 

[0098] Definition: Manipulating one or more subre- 
gions of the device such that the amount of light falling 
upon those subregions changes. 
[0099] Example: Consider a device used in a lecture 
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hall to take notes. When the room lights are turned on, 
the light sensors detect this and turn down the backlight 
to conserve energy. When the room lights are turned off 
(e.g. during a slide show) the light sensors detect this 
and turn up the backlight to increase viewability. 
[0100] Light sensors can range from thresholded bi- 
nary light detectors : to light pattern detectors, to full im- 
aging systems. Advanced techniques can include im- 
age analysis and recognition to identify objects or per- 
sons. 

HEAT 

[0101] Definition: Manipulating one or more subre- 
gions of the device such that the amount of heat applied 
to those subregions changes. 

[0102] Example: Consider a portable computer which 
has a stylus for entering text. By looking at the heat pro- 
file along the back surface of the computer, the compu- 
ter can detect whether it is being held with the left hand, 
the right hand, both hands, or neither hand, and update 
its interface accordingly. 

[0103] Thermal (heat) sensors can range from simple 
temperature sensors to sophisticated differential ther- 
mal mappers and thermal imagers. 

ELECTROMAGNETIC 

[0104] Definition: Manipulating one or more subre- 
gions of the device such that the electromagnetic spec- 
trum applied to those subregions changes. 
[01 05] Example: By analyzing the radio spectrum, the 
device can derive estimates as to its absolute spatial 
position, and use that to alter its functionality. 
[0106] Electromagnetic detection can include mag- 
netic compasses, radio detection, or GPS signal detec- 
tion. More advanced techniques can include electro- 
magnetic spectrum analysis and interpretation, such as 
roughly determining location based on available radio 
signals. 

VIBRATE 

[0107] Definition: Manipulating one or more subre- 
gions of the device by vibration. 
[0108] Example: Consider a device which displays 
textual information. When the user takes the device on 
the bus, the ambient vibration level sensed by the device 
changes, and the device increases the size o( the dis- 
played text to help the user compensate. 
[0109] This class ol environmental morphemes can 
include detection of intermittent contacts, low frequency 
rumblings, or acoustic level detection. More advanced 
techniques requiring greater processor power include 
maximum frequency identification, spectral analysis ol 
acoustic frequencies (enabling the device to distinguish 
background environmental noises from speech, for ex- 
ample), or even speech based identification of persons 



in the vicinity of a device. 

[0110] In addition to morphemes based on physical 
manipulation, spatial position, or sensed environmental 
factors, various morphemes based on cooperation be- 

5 tween multiple interacting devices are contemplated to 
be useful in practice of the present invention. To aid in 
understanding the diversity of contemplated spatial 
morphemes, Figure 28 illustrates multidevice mor- 
phemes arranged by an increasing level of possible 

io physical contact, and by increasing complexity of avail- 
able senseme tuples required to form or interpret a mor- 
pheme applied to a particular class of device. Beginning 
with a device having only rudimentary edge deformation 
functionality for supporting simple spatial sensemes, 

is and ending with complex deformable or embeddable 
devices that can be wrapped about each other, a defi- 
nition of a possible multidevice manipulation and typical 
function invoked by that multidevice manipulation of de- 
vices similar (but of course possibly more complex) to 

^0 that described in conjunction with Figure 1 is presented: 

TOUCH 

[0111] Definition: To move one or more subregions of 
2S a device such that they enter physical contact with a su- 
bregion of a second device, in any alignment and to any 
extent. Or, to take two devices so aligned and remove 
that alignment 

[0112] Example: Consider two portable computers 
30 350 and 351 , the first of which contains a database, and 
the second of which contains an IRDA port. When the 
user touches the first computer 350 to the second com- 
puter 351, the database is transmitted via the second 
computer's port. 

35 

MATCH 

[0113] Definition: To move one or more subregions of 
a device such that they enter physical contact with one 
40 or more subregions of a second device, with the subre- 
gion(s) of the first device and the subregion(s) of the 
second device being aligned along one or more edges. 
Or, to take two devices so aligned and remove that align- 
ment. 

45 [0114] Example: As illustrated in Figure 30, consider, 
multiple devices 360, 361 , and 362, which contain dif- 
ferent versions of the same basic database. When the 
user MATCHes the first device 360 to the second device 
361, followed by matching the third device 362 to the 

so matched first and second devices, their databases are 
reconciled (synchronized). 

STACK 

ss [011 5] Definition: To move one or more subregions ol 
a device such that they enter physical contact with a su- 
bregion of a second device, such that the first device is 
now located above, but physically adjacent to, the sec- 
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ond device. Or, to take two devices so aligned and re- 
move (i.e. unstack) that alignment. 
[0116] Example: As illustrated in Figure 31 . consider 
a set of devices 370, 371, and 372, each of which is 
displaying a frame ol video from a longer video se- 
quence. When the devices are stacked, the ordering ol 
the stacking specifies an order for the video editing, and 
a single composite video is now produced. 

TILE 

[0117] Definition: To move one or more subregions ol 
a device such that they physically contact with a subre- 
gion of a second device, such that the first device and 
second device now form a single seamless spatial unit 
or to take two devices so aligned and remove that align- 
ment. TILE is a special case of MATCH. 
[0118] Example: As illustrated in Figure 32, consider 
a set of devices 380, 381 , 382, 383, each of which can 
independently display a portion of a large photograph. 
When the devices are tiled, each device displays the 
portion of the photograph appropriate to its current rel- 
ative position in the tiled grid. 

RELATIVELY ALIGN 

[0119] Definition: To move one or more subregions ol 
a device such that they engage in a particular spatial 
relation to one or more other devices, where said devic- 
es are not touching. 

[0120] Example: As illustrated in Figure 33, consider 
a set of devices 390. 391 , 392, and 393 which are dis- 
playing a multi-page document. Whichever device is 
presently placed at the far left (device 390) displays the 
table of contents, whichever one is presently placed at 
the far right (device 393) displays the index, and the oth- 
ers display pages according to their respective loca- 
tions. As different devices can have different display ca- 
pabilities, moving them about can alter the document 
display. For example, if only one of the devices has a 
color display, when it is moved from second position to 
third position then (a) the device which was in third po- 
sition, displaying page #2, now displays page # 1, and 
(b) the color display, which was displaying page #1 , now 
displays page #2 in color. 

WRAPPING/EMBEDDING 

[0121] Definition. Manipulating one or more subre- 
gions of a device such that these subregions spatially 
occlude or are spatially occluded by some portion of the 
second device. 

[0122] Example: As illustrated in Figure 34, consider 
a first device 400 which contains the infrastructure to 
support the filtering of email. Consider a second set of 
devices 401 and 402 which implement particular email 
filters. When the user physically embeds device 401 (or 
402) into the first device 400, by which the first device 



400 now wraps the second device 401 (or 402), the par- 
ticular email filter supported by the second device is ac- 
tivated. 

[0123] As those skilled in the art will appreciate, com- 

5 binations of any of the foregoing described morphemes 
based on physical manipulation, spatial position, envi- 
ronmental conditions, or multiple interacting devices 
can be extended by participation in a morphemic "sen- 
tence". A sentence is defined as a sequence of one or 

10 more temporally disjoint morphemes. Typically, be- 
tween about 1/1 0th of a second and 2-3 seconds suffic- 
es to distinguish morphemes within a sentence. Of 
course, in some circumstances and indefinite time peri- 
od may elapse. The sentence level allows definition of 

is a physical manipulatory grammar by appropriate choice 
of a morpheme sequence, and corollary rules govern- 
ing, for example, use of active (verb like) morphemes, 
naming (noun) morphemes, or connectors. Just as the 
position and relation of words in a sentence define the 

20 sentence's meaning (e.g. "horse chestnut" is not the 
same as "chestnut horse"), similarly the position and re- 
lation of morphemes in a manipulatory sentence define 
the sentence's meaning. For example, in a communica- 
tion mode a FLICK followed by a WHACK could mean 

25 "transfer data and erase the local copyV while a WHACK 
followed by a FLICK means "power on the device and 
transfer data". In other settings, a FLICK or a WHACK 
could mean something entirety different. To better un- 
derstand construction of morphemic sentences, the fol- 

30 lowing examples are discussed: 

DATA TRANSFER SENTENCE 

[01 24] Consider a device which can transmit some or 
35 all of its information to another device. Furthermore, this 
transmission can be done either unencrypted, or en- 
crypted (to increase security). Furthermore, this trans- 
mission of a text+graphics document can either include 
the graphics, or omit them (to save time). Suppose that 
40 the user wishes to perform the command "Transmit the 
information in document A, encrypted, omitting graph- 
ics, to machine B". Then a gestural sequence (morphe- 
mic sentence) to support this could be: 

45 DEPRESS - the user presses on a displayed rep- 
resentation of A ; indicating that A is to be selected 
for an upcoming operation 

FLICK - the user flicks the device in the direction of 
device B, indicating that the operation is alransmis- 

50 sion to B 

FOLD - the user folds the top quarter of the device 
over the lower three-quarters, indicating that the 
transmission is to be encrypted. 
TWIST - the user twists the device about its central 

55 axis, indicating that the data is to be "wrung out", i. 
e. the graphics are to be omitted 
SQUEEZE - the user squeezes the device, indicat- 
ing that they are confirming that the operation is to 
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proceed. 

[0125] Note that none of these gestures, in isolation, 
performs an act - the "ensemble" of temporally separat- 
ed morphemes must be interpreted in order to form the 
complete action. 

DRAWING MODIFICATION SENTENCE 

[0126] Consider a device which displays geometric 
shapes for user manipulation. Furthermore, one of the 
manipulations supported is to resize (or rescale) a 
shape. Furthermore, suppose that this resizing can be 
done either aliased 0 a 99y) or anti-aliased (edges are 
smoothed). Suppose that the user wishes to perform the 
command "Resize shape A by 120%, about the X axis 
only, using anti-aliasing'. Then a morphemic sentence 
to support this could be: 

DEPRESS- the user presses on a displayed repre- 
sentation of A, indicating that A is to be selected for 
an upcoming operation 

STRETCH - the user stretches some portion of the 
device, indicating that the operation is to be a re- 
size. When the user starts stretching, a portion of 
the status display displays "100". The user contin- 
ues to stretch until the status display reads "120" 
RELIEF-MAP - the user "scores" the display by 
making a horizontal line of depression, indicating 
that the operation is to take place only about the 
horizontal (X) axis. 

DEPRESS - a circular thumb stroke is made in a 
different area of the device, indicating that anti- 
aliasing (smoothing the edges) is to be performed. 

DATABASE PRESENTATION SENTENCE 

[0127] Consider a device which contains various per- 
sonal information databases, such as a list of phone 
numbers, a list of addresses, and a calendar. Suppose 
that the user wishes the most appropriate of those da- 
tabases displayed. Then a gestural sequence to support 
this could be: 

SPATIAL LOCATION - the user carries the device 
such that it is spatially proximate to either the tele- 
phone, the address book, or the refrigerator (where 
the family calendar is displayed), whichever is ap- 
propriate. 

DEPRESS - the user touches the device to activate 
the operation. The device now displays the person- 
al information appropriate to that location. 

DATABASE RETRIEVAL SENTENCE 

[0128] To extend the foregoing example of a database 
presentation sentence, consider two computers which 
contain calendar databases. If the user wishes to syn- 



chronize the calendars, a suitable gestural sequence to 
support this could be: 

SQUEEZE - the user squeezes the device to acti- 
5 vates its gestural recognition capabilities 

ORBIT - the user orbits the device about the surface 
of the other device 3 times, indicating a desire to 
only match data for the next 3 weeks. 
MATCH - the user matches the edge of the device 
10 to the edge of the calendar, indicating a desire to 
"match" contents between the two devices. 

PRINTER/COPIER CONTROL SENTENCE 

15 [0129] Consider a device which can produce paper 
copies of documents. Suppose that the user wishes to 
tell such a device to produce a stapled, two sided copy 
of document A, enlarged to the next greater size. Then 
a gestural sequence to support this could be: 

20 

DEPRESS - the user presses on a displayed rep- 
resentation of A : indicating that A is to be selected 
for an upcoming operation 
RIP - the user introduces a spatial discontinuity into 
2S a portion of the device : indicating that the upcoming 
operation is to be a copy ("carrying away" some of 
the data). 

PINCH - the user pinches the upper left corner of 
the device, indicating that the copies are to be sta- 
30 pled. 

SQUEEZE - the user presses on the front and back 
of the device, indicating that the copy is to be two- 
sided. 

STRETCH - the user stretches the device, indicat- 
es ing that the copy is to be an enlargement to the next 
greater size. 

FACE - the device typically has its paper emitter on 
the bottom, preventing users from making copies 
accidentally. By facing the device such that the 
40 emitter is on the side, the copying operation is initi- 
ated. 

LIGHT BASED CONTROL SENTENCE 

45 [0130] Consider a device which can display docu- 
ments. Suppose the user is using the document while 
seated on a train, and wishes the document to display 
itsetf with a backlight when the train enters a tunnel, and 
wishes the document to display itself in a larger font 
50 when the train rumbies over rough tracks. Then a ges- 
tural sequence to support this could be 

SQUEEZE - the user squeezes on the device : indi- 
cating that a loss ol light is to be compensated tor 
55 by a backlight. 

LIGHT - as the train enters a tunnel, the LIGHT ges- 
ture is made, and the device turns on the backlight: 
WHACK - the user briskly raps the device against 
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the palm of their hand, indicating that their desired 
prelerence for adjusting hard-to-read documents is 
to increase the font size. 

VIBRATE - as the train goes over a bridge, the vi- 
bratory gesture is sensed. Because of the position 
of this VIBRATE morpheme (after the preceding 
WHACK gesture) in this morphemic sentence, the 
device now increases the font size on the displayed 
text. 

LIGHT - the user puts the device into his suitcase, 
making the LIGHT gesture. In this context (with no 
SQUEEZE before it), the LIGHT gesture causes the 
device to power-off its display. 
To better appreciate utility and construction of de- 
vices in accordance with the present invention, sev- 
eral examples of devices are now described: 

PORTABLE COMPUTER WITH SQUEEZE AND TILT 
CONTROL 

[0131] A handheld portable computer 500 (e.g. a 
3Com® PalmPilot®) capable of being fitted with deform- 
able, pressure sensitive edging 504 is schematically il- 
lustrated in Figures 35 and 36. The computer 500 sup- 
ports a name and address software application, provid- 
ing a user viewable name-address entry field on display 
503. In this embodiment a user can squeeze the de- 
formable, pressure sensitive edging 504 (squeeze ar- 
rows 507) of the computer 500. In response, the name 
and address software application causes the display 
503 to animate by slowly incrementing (scrolling) 
through the name listf rom "A" towards "Z" entries. When 
the user squeezes edging 504 again, the software ap- 
plication stops the scrolling animation. Scrolling func- 
tionality is further enhanced by the use of a tilt sensor, 
which allows the computer's behavior to mimic conven- 
tional rotatable address books. If the computer 500 is 
tilted away from the 45 degree angle at which someone 
might typically hold it, the scrolling rate is increased. In 
this application, the closer the computer 500 was tilted 
towards the user (as indicated by arrow 506 of Fig. 36), 
the faster the scroll rate toward "Z\ However, if a user 
tilted the computer 500 back past the neutral 45 degree 
position (as indicated by arrow 506 in Figure 36) ; the 
animation would move backwards with a velocity related 
to the magnitude of tilt. In this way it was possible for a 
user to search for items in a long list in a very natural 
way, while only using one hand. 
[0132] In an alternative mode schematically illustrat- 
ed in Figure 37, scrolling speed can be completely con- 
trolled by pressure. The greater the squeeze pressure 
(arrows 537), the faster the list scrolls. Release of the 
applied pressure causes the scrolling to halt. In this al- 
ternative user interface strategy, application tilt (as indi- 
cated by orthogonal tilt arrows 530 and 532) could be 
used to change the direction of the scrolling through the 
list, allowing a user to search portions of a large two di- 
mensional data set (schematically illustrated as a data 



plane 520) that is not entirely visible either horizontally 
or vertically on display 503. By simply tilting the display 
503 of computer 500 as if it were a window through 
which the data plane 520 can be viewed, any particular 
5 portion of the data plane (for example, data subset 524) 
can be viewed. As will appreciated, in both the loregoing 
modes the speed of scrolling, the specific neutral tilt an- 
gle, and required pressures to initiate scrolling changes 
can be adjusted to fit a particular user. 

10 

PORTABLE COMPUTER WITH HANDEDNESS 
DETECTION 

[0133] Pressure sensors have been added to aug- 
is ment a conventional keyboard 551 enabled user inter- 
lace to a hand holdable Windows® CE class computer 
550 (i.e. a Cassio® Cassiopia®) schematically illustrat- 
ed in Figures 38 and 39. In this embodiment, user hand- 
edness was determined by using pressure sensors po- 
20 sitioned on a right back-edge and a left back-edge of 
computer 550. User studies have found that the differ- 
ence in pressure between the right and left side gave a 
direct indication of handedness of a user. As respective- 
ly illustrated in Figures 38 and 39, handedness was 
25 used to justify formatted text 554 to the left (Fig. 38) or 
right (Fig. 39), thereby allowing more space 555 on dis- 
play 553 for an electronic annotation pen to be used to 
mark-up the text. 

[01 34] For both the embodiment of the invention iltus- 
30 trated by Figures 35-37, and the foregoing embodiment 
illustrated in Figures 38 and 39, material deformation of 
a spongy, elastic, or otherwise deformable material 
must be measured. Although various techniques can be 
used to measure material deformation, including those 
35 based on imaging or fluid volumetric changes, one par- 
ticularly useful technique is based on the use of pres- 
sure transducers. Commercially available sensors 
measure pressure (indicative of material deformation) 
by converting a pressure change to a change in electri- 
40 cal characteristics. For example, inexpensive sensors 
that change resistance in response to pressure can be 
obtained in a variety of shapes and sizes, including pa- 
per thin sensors, and easily bendable sensor strips. 
Sensors of this kind can be customized to any particular 
45 shape or form that a gestural Ul might require. The 
change in resistance is usually linearly related to pres- 
sure : with the sensor generally being placed in a poten- 
tial divider network to model the pressure as a change 
in potential. For a practical circuit the resulting signal 
so needs to be amplified, buffered and translated such that 
the change in value from minimum pressure to maxi- 
mum pressure spans a useful range. The modified sig- 
nal can now be fed into an analog to digital converter 
(ADC) to produce a digital representation of pressure. 
55 An 8-bil ADC can typically be used for most applications, 
however, if greater sensitivity to pressure changes is re- 
quired, a higher resolution ADC (e.g 1 6-bit ADC) can be 
used. As will be appreciated, the ADC could be memory 
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mapped into the processor's address space as a periph- 
eral, or alternatively supplied as a retrofitted pressure 
interface to existing computers capable of benefiting 
from this system. Since an RS232 connection is an in- 
terface that is almost universally available on portable 
computers, one strategy is to have the parallel output ol 
the ADC converted into a serial RS232 frame using a 
serializer such as a UART and then level-shift and buffer 
the signal as specified by the RS232 standard. At the 
computer end of the serial interface, another level-shift- 
er and UART, the output of which is readable by the 
processor, performs the serial-to-parallel conversion. 
[01 35] As described with reference to Figure 40, in re- 
alizing a working system a microcontroller 564 (a Sig- 
netics 87c752) with an ADC built in to combine many of 
the I/O tasks in a single chip can be used in combination 
with level shifter 566 (a MAX3223). This approach has 
the advantage that intelligent processing of the input sig- 
nal is possible with software. Tilt measurement was pro- 
vided by a tilt sensor 567 connected to a buffer/amp 562 
to supply an analog signal to microcontroller 564. The 
pressure measurement can also be encoded within a 
protocol across the serial link. This particular microcon- 
troller 564 has five ADC inputs, but by making use of 
eight digital control lines, it is possible to use only one 
ADC input and one buffer amplifier 561, to measure up 
to eight pressure points with pressure sensors 565. This 
is achieved by using the control lines to select only one 
sensor at a time and take a reading for each using a 
single input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory. The 
design is practical because the microcontroller can take 
measurements and make analog to digital conversions 
at a rate far higher than is needed to communicate with 
a host computer 569. 

[0136] For scrolling or handedness based software 
applications, 16 levels were determined to be adequate 
for representing a pressure measurement. In order to 
have a high data throughput to the host computer 569, 
each measurement was encoded in one byte of an 
RS232 frame such that the four lowest bits were the 
pressure representation and the highest four bits were 
the sensor ID. Each frame of the RS232 data was there- 
fore completely self contained. Of course any protocol 
that limits itself to a specific number of devices in its ad- 
dress space will eventually have the problem that some- 
day applications might be designed that need to refer- 
ence far more devices than are supported. The solution 
used in this protocol is to reserve the sensor ID number 
15 as a special value that can extend the semantics of 
the encoding to include an arbitrary number of bytes rep- 
resenting a sensor or value. For the described software 
applications, the commonly used RS232 frame format 
(1 start, 8 data no parity, 1 stop bit at a baud rate of 9600) 
was chosen. 

[0137] In operation, the host computer 569 deter- 
mined handedness by utilizing information about the 
current pressure exerted on two subregions - one, on 



the back of the device, occupying roughly the left half, 
and the other, also on the back of the device, occupying 
the right half. The current pressure values were convert- 
ed from analog to digital form, such that a digital value 
5 of 0 (zero) represents no pressure, and a digital value 
of, for example, 15 represents maximum pressure. The 
detection circuitry then proceeds: 

if (left_sensor is high AND right_sensor is high) then 
io conclude that the user is gripping the device 

with both hands 

else if (lelLsensor is high AND right_sensor is low) 
then 

conclude that the user is gripping the device 
is with the left hand 

only 

else if (!ett_sensor is low AND right_sensor is high) 
20 then 

conclude that the user is gripping the device 
with the right hand 

only 

25 

else if (left_sensor is low AND right_sensor is low) 
then 

conclude that the user is gripping the device 
with neither hand 

30 

Also ; to optimize communication, pressure values are 
only sent when they change. In order to account for jitter 
and error in the pressure sensors, a sensor is only con- 
sidered 'high* if its value is higher than some minimum 
35 threshold (e.g. '2' on the 0 to 15 range). 

SCANNER/PRINTER/COPIER SUPPORTING EDGE 
DEFORMABLE DISPLAY 

40 [0138] A scanner/printer/copier device 570 is sche- 
matically illustrated in Figure 41. As seen in Figure 41 
(and in more detail in Figure 42) the device 570 supports 
a display 574 having a deformable edge 572 : with the 
combination roughly shaped like a piece of paper. In op- 

45 sration, a user can place a written document in the de- 
vice 570 for scanning. An electronic version of the 
scanned document is displayed (i.e. as text 575 in Fig- 
ure 42) on the display 574. By outwardly pulling the de- 
formable edge 572 as indicated by arrow 577, a user 

so can instruct the device 570 to resize the document be- 
fore printing or copying. Pinching opposing sides of the 
deformable edge 572 (arrows 578) can further instruct 
the device 570 to print double sided copies. As will be 
appreciated, various other morphemes can be used to 

55 provide a simple interface tor interacting with device 
570. 
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TILEABLE AND STACKABLE PORTABLE DISPLAYS 

[0139] Multiple autonomous display tiles having an 
onboard display controller and at least one surface con- 
sisting substantially of a conventional display are partic- 
ularly useful for the practice of various aspects of the 
present invention. Such tiles can be Interconnected in 
response to various morphemes such as TOUCH, 
FLICK, RELATIVELY ALIGN, or WHACK, or even can 
be operated without substantial user mediated morphe- 
mic input in certain situations. 
[0140] Advantageously, tile positioning can be used 
as an interlace specifier in and of itself. For example, 
each display tile can contain a video segment in inde- 
pendent memory. Shuffling or reorganizing the tiles can 
allow users to physically manipulate the sequence of 
video segments to affect a physically manipulatable vid- 
eo editing system. Using the card analogy, tiles can be 
used to re-sequence documents : pages within a docu- 
ment, audio annotations, voice mail, or other temporal 
media contained within the tiles. The resultant sequence 
can then be played as a whole by using the tiled struc- 
ture as a unit. 

[0141] For purposes of the present invention, display 
tile array configurations can be categorized as follows: 
[0142] Close-packed display tile array 600 (Figure 43) 
in which tiles 602 are arranged to span a surface 610 in 
closely abutting, but not overlapping relationship, such 
that the continuous display area is maximized (i.e. 
seamlessly). As will be appreciated, the tiles could 
themselves form a freestanding continuous surface, or 
they could be disposed on a table or other suitable sup- 
port. Each of the tiles 602 supports a display 604 sized 
to substantially cover a front surface of each tile 602. In 
certain embodiments, a back surface of each tile 602 
can also support a display. Advantageously, this would 
allow for creation of free standing displays with images 
visible on the front and back. The surface 610 can be a 
plane, a sphere, or any arbitrary shape that permits til- 
ing. 

[0143] Loose-packed display tile array 620 (each tile 
622 having a display 624 as seen in Figure 44) in which 
tiles 622 are slotted into a lattice pattern (indicated by 
dotted lines 625). Each tile can be considered to be sit- 
uated in a regular bounded lattice slot (namely, the lat- 
tice slot that contains a defined center of each of the 
tiles) within the lattice where the dimensions of the 
bounded slot are no more than a few times the maximum 
dimension of the tile, and there is no more than one tile 
in any slot. Within the lattice regions any of the tiles can 
be positioned arbitrarily and still retain the same inter- 
relationship with the group. It is still possible for tiles to 
touch one another at the boundaries of two or more lat- 
tice slots, however, this is not a requirement. 
[0144] Free-format display tiles 630 (each tile 632 
having a display 634 as seen in Figure 45) are similar 
to loose-packed display tiles 620 where the size and 
shape of the lattice slots may vary freely (e.g. the dimen- 



sion of a lattice slot may be many times the maximum 
dimension of any of the participating tiles). The only con- 
straint on the arrangement is that there must be no am- 
biguous relationships about the relative connectivity of 

5 the each of the tiles 632. That is to say, a tile that is to 
display the next piece of information to one side of an- 
other tile must be unique and not be confused with the 
task of another tile in the tiling lattice. 
[01 45] 3D display tiles (packable display tiles) are cre- 

io ated by extending the three foregoing display tile cate- 
gories. However for close packed tiling of display tiles, 
packed devices in the center of a 3D structure would not 
be available as the user interlace. This may not matter 
as the surface of the 3D shape will expose a area that 

15 will have unique affordances for some classes of appli- 
cation. For instance, cubic tiles packed into the shape 
of a large cube can display, using the six faces of the 
larger cube, the various projections that could be ren- 
dered by viewing a 3D CAD drawing from each degree 

20 of freedom. 

[0146] As will be appreciate display tiles need not 
be recti-linear but can be hexagonal, circular, or of arbi- 
trary shape and size. The size of tiles need not be con- 
stant for all tiles within a larger structure. Tiles need not 

25 be aligned but may require proximity to indicate adjoin- 
ing points or edges. 

[01 47] Tiles need not be in physical contact to define 
they are joined in a group activity but instead this may 
be a programmed function. The connectivity can also 
30 be discovered through a radio network from a coordi- 
nating server or from a distributed algorithm that draws 
in as many computers as necessary for a task : using the 
wireless network to negotiate the resources it needs. 
The tiled computers may also be connected by wired 
35 networking systems, although in cases where the topol- 
ogy might need to rapidly change, this is not as desirable 
as a wireless system. An example of a wired networking 
system of this type is a system that uses the internet to 
include many computers in a single task, one in which 
40 each computer knows its relative position even though 
they may be separated by rooms or in the extreme cases 
buildings, cities or countries. However, in general the 
most uselul case of display tiling is when the tiles are 
proximate enough that they can all be viewed by one 
45 person to create a display medium whereby the viewing 
experience is an enhancement over using a single dis- 
play. Accordingly, tiles can operate as either a single 
larger contiguous structure or they can retain individual 
properties and independent functions or a combination 
50 of both. (e.g. , jumbo-tron like function, 1 2x1 2 small dis- 
plays as used in a TV studio , or picture in-picture fea- 
tures found in commercial TVs or editing suites). 
[0148] Depending on the type of packing, various 
schemes can be used to allow lor permanent, intermit- 
55 tent, or even one-time communication between display 
tiles. For example, close packed tiling can make use of 
wired connectivity between computers or could use a 
variety of wireless or optical communication technolo- 
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gies. In the case of wired connectivity: edge mounted 
conventional plug and socket connectors may be used 
to create a rigid tiled array. Plug and socket systems 
lend themselves to parallel connections for bulk and 
high-speed data transfer. They also provide a conven- 
ient method of power distribution, which can allow for 
one of the tiles providing a power source that supports 
the rest of the array. A plug and socket connection be- 
tween tiled computers 692 in a tiled computer array 690 
(with both data and power transfer shown) is schemat- 
ically illustrated by lines 695 in Figure 48. 
[0149] The exact design and locations ol electrical 
connectors depends on the intended use, and subse- 
quently, the shape of tile components. For applications 
where large seams between displays are acceptable, 
simple rigid connectors could be attached to the center 
of each edge, providing connectivity to all the surround- 
ing tiles. Other applications might require a more com- 
plex design. For example ; applications which require 
high-quality information display on a seamless array 
surface (i.e. tiled blueprints), might employ spring-load- 
ed contacts on the four edge connectors. The spring 
mechanism allows all array connections to be made be- 
low the display surface, while tiles can still be inserted 
and removed from the interior of the array. Removal of 
tiles could be triggered by some host signal which actu- 
ates a release of the spring and pops the tile out of the 
array. 

[0150] Serial connectivity can also be used in a close 
packed arrangement such as that illustrated in Figure 
43. It has the advantage that fewer connections have to 
be made and in practice it might be more reliable. How- 
ever the net bandwidth between tiles will be less than a 
parallel system. Serial communication lends itself to op- 
tical and wireless systems thus removing the need for 
any physical connections. For optical technologies 
alignment of the transmitter and receiver is still impor- 
tant although the clever use of light pipes and lens cap- 
ture techniques can introduce more flexibility. Wireless 
systems can use many different bands of the EM spec- 
trum (kHz ; MHz. GHz), utilizing a variety of modulation 
techniques (amplitude modulated, frequency modulat- 
ed, or those based on code division muftiple access) and 
operate at a range of transmitter powers. There is no 
longer a need for direct alignment if the system is de- 
signed with suitable communication tolerances. The 
transmitter range plays a crucial part in the design. If 
there is only enough power in an emitted signal to be 
picked up within a few millimeters of a tile edge, then 
the signals will be isolated, the topology will be defined 
by the physical connectivity and the complexity of de- 
signing the system to avoid interference from neighbor- 
ing signal sources will be minimized. However, an alter- 
native design is to use more powerful radios. In this case 
all tiles will be able to contact all other tiles and inter-tile 
connectivity needs to be defined by another parameter. 
Signal strength can be used or, more deliberately, infor- 
mation that relates the ID of a tile to a spatial map (per- 



haps held in one master tile) describing the position of 
all tiles in the tile array. In this system it is also necessary 
to minimize inter-tile interference. For digital packet-da- 
ta systems that operate at the same frequency, carrier 

5 sense multiple access (CSMA-CD or CSMA-CA) sys- 
tems are well known techniques to solve this problem. 
Other solutions involve tiles using different frequencies, 
with frequencies reused depending on the power of the 
transmitters. This is the technique used by traditional 

io cellular telephones. Yet another approach is to use 
Code Division Muftiple Access (CDMA) that relies on the 
overlaying of signals in the same region of the EM spec- 
trum, a technique known as spread spectrum modula- 
tion. 

15 [0151] For loose packed display tiles, the wireless 
techniques described above in connection with close 
packed tiling generally become essential to the imple- 
mentation. However, a special case of loose packing ex- 
ists in which each edge of a tile does make contact with 

20 every other surrounding tile, except that it may only be 
a single point of contact and not at a accurately defined 
place. A wired version of this system can be built in 
which the entire edge of each tile is a serial connection 
including one of the two vertices that define the edge. 

25 Communication in each direction can be achieved by a 
variety of commercially available techniques, including 
use of a one-wire interface (plus a ground) for bidirection 
communication between tags and a reader. Note that 
the ground connection for a tile arrangement can be de- 

30 rived by sharing a common ground connection through 
the surface the tiles are laid out on. For example, the 
surface could be made of a metal sheet. The system 
may be lurther enhanced by ensuring that the edge con- 
tact is made of a magnetic material and the vertices 

35 have a magnet embedded in the end. Such an arrange- 
ment ensures that there will be a good electrical contact 
between the transmitter and the receiver. 
[0152] The receiver can also derive power from an 
electrically transmitted signal by bridge-rectifying it and 

40 storing the collected charge in a capacitor for use by its 
own electronics. Thus, power distribution can also be 
included in a one-wire interface. In this way flexible con- 
nectivity can be achieved to support the rapid and con- 
venient rearrangement of tiles in which only the mini- 

45 mum amount of care needs to be taken in setting up 
connectivity. 

[0153] Loosely packed tile displays such as illustrated 
in Figure 45 may require that the display surfaces use 
a best elforl algorithm to present a unified display with 

50 all the sections of the displayed image bearing the cor- 
rect spatial arrangement to each other even though, in 
the case of rectangular tiles, they might not be regis- 
tered vertically or horizontally and have an offset angle 
relative to each other. In order to implement the desired 

55 tite display algorithm, not only is the relative arrange- 
ment of tiles important but the exact offset (distance and 
angle) from each other is also important. There are sev- 
eral methods that are suitable for automatically deter- 
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mining offsets between loosely packed tiles. For exam- 
ple, as illustrated with respect to Figure 46 (showing 
communicating tiles 652 and 654, with respective dis- 
plays 651 and 653), optical encoding 660 along an edge 
can be used to identify tile orientation. It is possible to 
use a binary coded optical pattern along an edge which 
is regular and encodes the distance from a vertex at any 
point. A tile that abuts, or is relatively aligned : can read 
this pattern using optical sensors 658 and 659 and de- 
termine the display offset in the direction of the edge. 
Alternatively, as illustrated by Figure 47 (showing com- 
municating tiles 672 and 674), it is possible to use radio 
based techniques that rely on signal strength triangula- 
tion. Each vertex 675 or 676 of a tile 672 can contain a 
wireless transmitter and a receiver. If these vertices 
transmit a short characteristic wireless signal at well 
known times, a nearby tile 674 can use the receivers 
situated at its own vertices 660, 6B1, 682, and 683 to 
triangulate the position of each transmitting vertex rela- 
tive to them by measuring the relative delay by which 
the signal was received. After two of the vertices of the 
original transmitting tile 672 have signaled, the adjoining 
tile 674 can determine its exact position and orientation 
within their local region of a tile array. The transmitting 
and receiving tiles can now swap roles, with the result 
that both tiles will know their relative position. This proc- 
ess can be extended across the tile array. 
[0154] Free format tiling differs from loose-packed til- 
ing in that there are no proximity, or regular format, con- 
straints. To demonstrate how a free format tiling system 
might work, the following example is described. Imagine 
a number of laptop computers each fitted with a Global 
Positioning System (GPS) and a radio modem. Each 
laptop can determine its position (with acceptable error) 
and communicate it to all the other laptops by contacting 
them through the radio modem. After some period of 
time the laptops will all know their relative location and 
absolute location. If any computers changes its location, 
it can update the local neighbors to ensure there is 
enough understanding in the array to consider the com- 
puters are in a known tiling configuration even though 
they are not close to each other and in fact they may be 
in different geographic regions. An application that 
might use this free format tiling system is one that wish- 
es to ensure information is being correctly sent in a uni- 
form and dispersed fashion across a very large area. 
For example, the information that each laptop computer 
receives could be instructions to release an amount of 
a pesticide used to control an insect that does damage 
to commercial crops. If the pesticide is released in too 
high a concentration in any locality, it may be hazardous 
to human health The tiled approach allows the comput- 
ers to roam (e.g. in the back of a truck), displaying in- 
formation as to the type and concentration of pesticide 
that is to be released given their current relative prox- 
imity. 

[0155] Various conventional algorithms can be used 
to support distribution of information between autono- 



mous tiled displays. These algorithms assume a system 
by which there is a master controller generating data to 
be displayed. There is also a large array of tiled com- 
puters that the system will use to display visual data and/ 

5 or process information. Each computer in the tiled array 
contains a unique ID. It is the job of the master to split 
the data into pieces that each computer can display and 
for this information to be packaged along with the ID of 
the target computer. The algorithms belowdescribe how 

w the information travels from the master to a destination 
display tile. 

Daisy chain routing 

is [0156] Display tiles are arranged to have a logical 
connectivity with each other so that each one only trans- 
mits information to the next in a predefined line. The 
computers are said to be daisy chained to each. Any 
information sent to the start of the chain contains an ID 

20 and the first computer in the chain compares it to its own 
ID. If it matches : it acts on the data. If not, it sends the 
data on to the next computer in the chain until it finds its 
destination 

2S N-ary routing 

[0157] In N-ary routing the path to the destination is 
contained in the ID of the device. A simple routing ex- 
ample is schematically illustrated with retrence to the di- 
30 rection arrows 695 in Figure 48, which shows quater- 
nary routing in physically connected tileable displays. In 
quaternary routing, an array is conceptually arranged as 
a quaternary tree with each node having an input and 
three outputs. In this system each pair of bits of the ID 

35 contains a routing command. A 0 indicates send the 
packet to the first output, a 1 indicates the second out- 
put, a 2 indicates the third output, and 3 indicates no 
further transmission. There is also a count that gets dec- 
remented by each node to tell successive nodes the bit 

40 number that is currently being considered and when the 
packet has reached its destination In this way packets 
are forwarded from node to node with a simple choice 
at each stage until they reach their target display. As will 
be appreciated, it is possible to design N-ary systems 

45 with more than three outputs (a power of two is usually 
convenient for implementation (e.g. 4, 8 , 16 ...)). 

Flooding 

50 [0158] Flooding has no predefined routing structure. 
The computer that takes the first packet from the master 
checks to see if it has the correct ID. It not, the packet 
is sent out on all links to which it has not yet sent or 
received that packet. The result is a flood of copies of 
55 the packet across the array eventually reaching its des- 
tination. The packets must also have a maximum hop 
count to ensure they eventually are removed from the 
system. The disadvantage of this approach is that many 



20 



37 



EP 0 899 650 A2 



38 



more tiles are burdened with the processing of unwant- 
ed data than in the previous two schemes, which may 
impact the overall efficiency of the system. 

Hot Potato 

[0159] The Hot Potato algorithm is similar to the 
Flooding algorithm, except a retransmitted packet is 
sent out on only one output that is either chosen ran- 
domly or is the least busy. The process stops when the 
packet reaches the correct tile. The time that a packet 
takes to reach its destination is not deterministic. 



Claims 
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A device supporting a manipulatable user interface, 
the device comprising: 

a feedback module (33) for presenting informa- 
tion related to a data structure, 
a processor (24) for controlling the feedback 
module (33) and the data structure, and 
a detector (20) for determining handedness of 
a user, the detector being connected to the 
processor (24) for modifying the displayed data 
structure based on handedness of a user. 

A method for inputting information to a device con- 
nected to a deformable piece, the method compris- 
ing the steps of: 



1. A device (10) supporting a manipulatable user in- 
terface, the device comprising: 

a feedback module (33) for presenting informa- 20 
tion related to a data structure, 
a processor (24) for controlling the feedback 
module (33) and the data structure, and 
a deformable piece (20) including multiple su- 
bregions, the deformable piece attached in a 25 
vicinity to the feedback module, with the de- 
formable piece contacting at least one sensor 
(22) that monitors positional changes within 
multiple subregions of the deformable piece 
(20), the at least one sensor (22) being con- 30 
nected to the processor (24). 

2. A method for inputting information to a device con- 
nected to a deformable piece, the method compris- 
ing the steps of: 35 

manipulating the deformable piece to provide a 
first morpheme input to the device, the first mor- 
pheme input normally triggering a first default 
action by the device, 40 
asynchronously manipulating the deformable 
piece to provide a second morpheme input to 
the device, with the second morpheme input 
converting the normally triggered first default 
action to a second action. *s 



spatially rotating device to provide a first mor- 
pheme input to the device, the first morpheme 
input normally triggering a first default action by 
the device, and 

asynchronously manipulating the deformable 
piece to provide a second morpheme input to 
the device, with the second morpheme input 
converting the normally triggered first default 
action to a second action. 

6. A plurality of tileable devices for transferring data, 
comprising: 

a first device having a processor and a first 
wireless communication module for transfer- 
ring data, 

a second device having a processor and a sec- 
ond wireless communication module for trans- 
ferring data, 

a third device having a processor and a third 
wireless communication module for transferr- 
ring data, 

wherein the first device is connected in sub- 
stantially simultaneous wireless communica- 
tion with the second device and the third device 
to pass data based on spatial positions of the 
respective first device, second device, and third 
device. 



3. A device supporting a manipulatable user interface, 
the device comprising: 

a processor (24) and memory (26), and so 
a deformable piece (20) including multiple su- 
bregions, the deformable piece (20) attached 
in a vicinity to the processor (24) and memory 
(26), with the deformable piece (20) contacting 
a sensor mesh (22) that monitors positional 55 
changes within multiple subregions of the de- 
formable piece, the sensor mesh (22) being 
connected to the processor (24). 
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Fig. 10 



27 



EP 0 899 650 A2 




28 




29 



EP 0 899 650 A2 




30 



EP 0 899 650 A2 



CO 

-o 



TO 

i 



CD 

E 



o 



E 

I 

E 



£ E 

5 8 



o 

03 

5 



> 
c: 

CD 
O 



.2 

CD 



•e 

o 



cd 
E 

S 
o 

E 



c 

"i— 
o 



cx 
c: 

CD 



3 = 



CD 

-o 

O 



-5> 



c 

CD 

E 
c 
o 



CD 

k— 

a) 

">< 

CD 
O 

CD 
CD 

s 
g 



CD 



a 



AjixaidwoQ oiiuasuas 6ui$esjou| 



31 



EP 0 899 650 A2 




EP 0 899 650 A2 




EP 0 899 650 A2 




EP 0 899 650 A2 




EP 0 899 650 A2 




Fig. 26 
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Fig. 34 
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Fig. 37 
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Fig. 38 




Fig. 39 
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